System and methods for creation of learning agents in simulated environments

ABSTRACT

A system and methods for generating and applying learning agents in simulated environments, in which an agent simulation is selected, one or more agent goals are received, and agents are created which are individual instances of the agent simulation with each agent having at least one of the agent goals, wherein the agents are used in the execution of an environment simulation which dynamically changes based on the collective behavior of the agents.

CROSS-REFERENCE TO RELATED APPLICATIONS

Priority is claimed in the application data sheet to the followingpatents or patent applications, the entire written description of eachof which is expressly incorporated herein by reference in its entirety:

Ser. No. 15/882,180

Ser. No. 15/835,436

Ser. No. 15/790,457

Ser. No. 15/790,327

62/568,291

Ser. No. 15/616,427

Ser. No. 14/925,974

Ser. No. 15/141,752

Ser. No. 15/091,563

Ser. No. 14/986,536

62/568,298

Ser. No. 15/835,312

Ser. No. 15/186,453

Ser. No. 15/166,158

BACKGROUND OF THE INVENTION Field of the Art

The disclosure relates to the field of digital simulations, morespecifically to the field of simulated actors or agents with machinelearning in simulated environments.

Discussion of the State of the Art

Currently, it is possible for computers and servers to run rigid,unchanging, OS-specific and language-specifics simulation environmentswith pre-set agents or actors in them, to achieve a variety of results.There are issues with the current state of simulation engines and withthe generation of actors or agents in those engines, however. Currently,it is impossible to generate agents (which will be used interchangeablywith “actors” for the rest of this disclosure) which are capable ofmachine learning techniques inside of simulated environments, mostespecially simulated environments which are generated procedurally orstochastically and not hard-coded environments which are un-changingfrom one runtime to the next. Agents in simulated environments operateon hard-coded and static behaviors which are un-changing for the mostpart, and are not extremely useful to a variety of simulatedenvironments which may elucidate real-world situations or environmentswith machine learning.

What is needed is a system and methods for creation and execution oflearning agents in simulated environments, so that simulatedenvironments may be executed with agents operating in the environmentthat are capable of learning and evolving their behavior, to simulate anumber of real-world situations such as economic environments orbiological systems.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, in apreferred embodiment of the invention, a system and methods forgenerating learning agents in simulated environments. The followingnon-limiting summary of the invention is provided for clarity, andshould be construed consistently with embodiments described in thedetailed description below.

To solve the problem of static or hard-coded agents in simulatedenvironments, a system and method have been devised for agents whichutilize machine learning, inside simulated environments which may bestochastic and ever-changing in nature, as opposed to simulatedenvironments which remain essentially the same from one execution to thenext.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several aspects and, together withthe description, serve to explain the principles of the inventionaccording to the aspects. It will be appreciated by one skilled in theart that the particular arrangements illustrated in the drawings aremerely exemplary, and are not to be considered as limiting of the scopeof the invention or the claims herein in any way.

FIG. 1 is a diagram of an exemplary architecture of a business operatingsystem according to a preferred aspect.

FIG. 2 is a diagram of an exemplary architecture of an automatedplanning service cluster and related modules according to a preferredaspect.

FIG. 3 is a diagram of an exemplary architecture of a system for thecapture and storage of time series data from sensors with heterogeneousreporting profiles according to an embodiment of the invention.

FIG. 4 is a diagram of an exemplary architecture of a system for agentcreation for at least one simulated environment, but possibly several,across a network, according to a preferred aspect.

FIG. 5 is a method diagram illustrating agent creation for simulatedenvironments, on a server, according to a preferred aspect.

FIG. 6 is a method diagram illustrating agents acting in a simulatedenvironment and utilizing machine learning to evolve their behavior inthe simulated environment, according to a preferred aspect.

FIG. 7 is a method diagram illustrating agents in a simulated biologicalenvironment, according to a preferred aspect.

FIG. 8 is a method diagram illustrating agents acting in a simulatedeconomic environment, according to a preferred aspect of the invention.

FIG. 9 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device.

FIG. 10 is a block diagram illustrating an exemplary logicalarchitecture for a client device.

FIG. 11 is a block diagram showing an exemplary architecturalarrangement of clients, servers, and external services.

FIG. 12 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, a system and methodfor system and methods for creation and execution of learning agents insimulated environments.

One or more different aspects may be described in the presentapplication. Further, for one or more of the aspects described herein,numerous alternative arrangements may be described; it should beappreciated that these are presented for illustrative purposes only andare not limiting of the aspects contained herein or the claims presentedherein in any way. One or more of the arrangements may be widelyapplicable to numerous aspects, as may be readily apparent from thedisclosure. In general, arrangements are described in sufficient detailto enable those skilled in the art to practice one or more of theaspects, and it should be appreciated that other arrangements may beutilized and that structural, logical, software, electrical and otherchanges may be made without departing from the scope of the particularaspects. Particular features of one or more of the aspects describedherein may be described with reference to one or more particular aspectsor figures that form a part of the present disclosure, and in which areshown, by way of illustration, specific arrangements of one or more ofthe aspects. It should be appreciated, however, that such features arenot limited to usage in the one or more particular aspects or figureswith reference to which they are described. The present disclosure isneither a literal description of all arrangements of one or more of theaspects nor a listing of features of one or more of the aspects thatmust be present in all arrangements.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or morecommunication means or intermediaries, logical or physical.

A description of an aspect with several components in communication witheach other does not imply that all such components are required. To thecontrary, a variety of optional components may be described toillustrate a wide variety of possible aspects and in order to more fullyillustrate one or more aspects. Similarly, although process steps,method steps, algorithms or the like may be described in a sequentialorder, such processes, methods and algorithms may generally beconfigured to work in alternate orders, unless specifically stated tothe contrary. In other words, any sequence or order of steps that may bedescribed in this patent application does not, in and of itself,indicate a requirement that the steps be performed in that order. Thesteps of described processes may be performed in any order practical.Further, some steps may be performed simultaneously despite beingdescribed or implied as occurring non-simultaneously (e.g., because onestep is described after the other step). Moreover, the illustration of aprocess by its depiction in a drawing does not imply that theillustrated process is exclusive of other variations and modificationsthereto, does not imply that the illustrated process or any of its stepsare necessary to one or more of the aspects, and does not imply that theillustrated process is preferred. Also, steps are generally describedonce per aspect, but this does not mean they must occur once, or thatthey may only occur once each time a process, method, or algorithm iscarried out or executed. Some steps may be omitted in some aspects orsome occurrences, or some steps may be executed more than once in agiven aspect or occurrence.

When a single device or article is described herein, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described herein, it will be readily apparent that a singledevice or article may be used in place of the more than one device orarticle.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other aspects need notinclude the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should beappreciated that particular aspects may include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. Process descriptions or blocks in figures should beunderstood as representing modules, segments, or portions of code whichinclude one or more executable instructions for implementing specificlogical functions or steps in the process. Alternate implementations areincluded within the scope of various aspects in which, for example,functions may be executed out of order from that shown or discussed,including substantially concurrently or in reverse order, depending onthe functionality involved, as would be understood by those havingordinary skill in the art.

Definitions

As used herein, a “meta-model,” “meta model” or “metamodel” is a datastructure representing relationships between simulation models andsimulated environments using those models, as well as external orlocally based computer tools which can be used for computationalpurposes. A meta-model is able to allow simulated environments which usea given meta-model to use specific models for simulation which normallypossess strict limitations and functions, but use other simulationmodeling engines such as different physics engines, without changing thecontent or function of any simulation model or the engine running it.For example, a single meta-model might record a relationship between twophysics engines, one being a generic physics engine with components suchas rag-doll physics, jump speed for characters in a video game, bulletphysics, and destructible environment physics, and the other physicsengine being solely flight simulation. In such an example, themeta-model may specify that the flight simulation engine's physics maybe applied to objects that, in the generic physics engine, are recordedas “plane” objects, thereby bypassing the need to alter either engine,and allowing both to work as-is, with each other, through the use ofspecified relationships in the meta-model.

Conceptual Architecture

FIG. 1 is a diagram of an exemplary architecture of a business operatingsystem 100 according to an embodiment of the invention. Client access tothe system 105 for specific data entry, system control and forinteraction with system output such as automated predictive decisionmaking and planning and alternate pathway simulations, occurs throughthe system's distributed, extensible high bandwidth cloud interface 110,connected to the wider cloud 107 through use of a network includingInternet Protocol (IP) networks, which uses a versatile, robust webapplication driven interface for both input and display of client-facinginformation and a data store 112 such as, but not limited to MONGODB™,COUCHDB™, CASSANDRA™ or REDIS™ depending on the embodiment. Much of thebusiness data analyzed by the system both from sources within theconfines of the client business, and from cloud based sources, alsoenter the system through the cloud interface 110, data being passed tothe connector module 135 which may possess the API routines 135 a neededto accept and convert the external data and then pass the normalizedinformation to other analysis and transformation components of thesystem, the directed computational graph module 155, high volume webcrawler module 115, multidimensional time series database 120 with API'sor other programming wrappers 120 a and the graph stack service. Thedirected computational graph module 155 retrieves one or more streams ofdata from a plurality of sources, which includes, but is in no way notlimited to, a plurality of physical sensors, web based questionnairesand surveys, monitoring of electronic infrastructure, crowd sourcingcampaigns, and human input device information. Within the directedcomputational graph module 155, data may be split into two identicalstreams in a specialized pre-programmed data pipeline 155 a, wherein onesub-stream may be sent for batch processing and storage while the othersub-stream may be reformatted for transformation pipeline analysis. Thedata is then transferred to the general transformer service module 160for linear data transformation as part of analysis or the decomposabletransformer service module 150 for branching or iterativetransformations that are part of analysis. The directed computationalgraph module 155 represents all data as directed graphs where thetransformations are nodes and the result messages betweentransformations edges of the graph. The high volume web crawling module115 uses multiple server hosted preprogrammed web spiders, which whileautonomously configured are deployed within a web scraping framework 115a of which SCRAPY™ is an example, to identify and retrieve data ofinterest from web based sources that are not well tagged by conventionalweb crawling technology. The multiple dimension time series databasemodule 120 receives data from a large plurality of sensors that may beof several different types. The module is designed to accommodateirregular and high volume surges by dynamically allotting networkbandwidth and server processing channels to process the incoming data.Inclusion of programming wrappers for languages examples of which are,but not limited to C++, PERL, PYTHON, and ERLANG™ allows sophisticatedprogramming logic to be added to the default function of themultidimensional time series database 120 without intimate knowledge ofthe core programming as a wrapper or add-on 120 a, greatly extendingbreadth of function. Data retrieved by the multidimensional time seriesdatabase 120 and the high volume web crawling module 115 may be furtheranalyzed and transformed into task optimized results by the directedcomputational graph 155 and associated general transformer service 150and decomposable transformer service 160 modules. Alternately, data fromthe multidimensional time series database and high volume web crawlingmodules may be sent, often with scripted cuing information determiningimportant vertexes 145 a, to the graph stack service module 145 which,employing standardized protocols for converting streams of informationinto graph representations of that data, for example, open graphinternet technology although the invention is not reliant on any onestandard. Through the steps, the graph stack service module 145represents data in graphical form influenced by any predeterminedscripted modifications 145 a and stores it in a graph-based data store145 b such as GIRAPH™ or a key value pair type data store REDIS™, orRIAK™, among others, all of which are suitable for storing graph-basedinformation.

Results of the transformative analysis process may then be combined withfurther client directives, additional business rules and practicesrelevant to the analysis and situational information external to thealready available data in the automated planning service module 130which also runs powerful information theory 130 a based predictivestatistics functions and machine learning algorithms to allow futuretrends and outcomes to be rapidly forecast based upon the current systemderived results and choosing each of a plurality of possible businessdecisions. The using all available data, the automated planning servicemodule 130 may propose business decisions most likely to result is themost favorable business outcome with a usably high level of certainty.Closely related to the automated planning service module in the use ofsystem derived results in conjunction with possible externally suppliedadditional information in the assistance of end user business decisionmaking, the action outcome simulation module 125 with its discrete eventsimulator programming module 125 a coupled with the end user facingobservation and state estimation service 140 which is highly scriptable140 b as circumstances require and has a game engine 140 a to morerealistically stage possible outcomes of business decisions underconsideration, allows business decision makers to investigate theprobable outcomes of choosing one pending course of action over anotherbased upon analysis of the current available data. For example, thepipelines operations department has reported a very small reduction incrude oil pressure in a section of pipeline in a highly remote sectionof territory. Many believe the issue is entirely due to a fouled,possibly failing flow sensor, others believe that it is a proximalupstream pump that may have foreign material stuck in it. Correction ofboth of these possibilities is to increase the output of the effectedpump to hopefully clean out it or the fouled sensor. A failing sensorwill have to be replaced at the next maintenance cycle. A few, however,feel that the pressure drop is due to a break in the pipeline, probablysmall at this point, but even so, crude oil is leaking and the remedyfor the fouled sensor or pump option could make the leak much worse andwaste much time afterwards. The company does have a contractor about 8hours away, or could rent satellite time to look but both of those areexpensive for a probable sensor issue, significantly less than cleaningup an oil spill though and then with significant negative publicexposure. These sensor issues have happened before and the businessoperating system 100 has data from them, which no one really studied dueto the great volume of columnar figures, so the alternative courses 125,140 of action are run. The system, based on all available data, predictsthat the fouled sensor or pump is unlikely to be the root cause thistime due to other available data, and the contractor is dispatched. Shefinds a small breach in the pipeline. There will be a small cleanup andthe pipeline needs to be shut down for repair but multiple tens ofmillions of dollars have been saved. This is just one example of a greatmany of the possible use of the business operating system, thoseknowledgeable in the art will easily formulate more.

FIG. 2 is a diagram of an exemplary architecture of an automatedplanning service module and related modules 200 according to anembodiment of the invention. Seen here is a more detailed view of theautomated planning service module 130 as depicted in FIG. 1 . The modulefunctions by receiving business decision or business venture candidatesas well as relevant currently available related data and any campaignanalysis modification commands through a client interface 205. Themodule may also be used provide transformed data or run parameters tothe action outcome simulation module 125 to seed a simulation prior torun or to transform intermediate result data isolated from one or moreactors operating in the action outcome simulation module 125, during asimulation run. Significant amounts of supporting information such as,but not restricted to current business conditions, infrastructure,ongoing venture status, financial status, market conditions, and worldevents which may impact the current decision or venture that have beencollected by the business operating system as a whole and stored in suchdata stores as the multidimensional times series database 120, theanalysis capabilities of the directed computational graph module 155 andweb-based data retrieval abilities of the high volume web crawler module115 all of which may be stored in one or more data stores 220, 225 mayalso be used during simulation of alternative business decisionprogression, which may entail such variables as, but are not limited toimplementation timing, method to end changes, order and timing ofconstituent part completion or impact of choosing another goal insteadof an action currently under analysis.

Contemplated actions may be broken up into a plurality of constituentevents that either act towards the fulfillment of the venture underanalysis or represent the absence of each event by the discrete eventsimulation module 211 which then makes each of those events availablefor information theory based statistical analysis 212, which allows thecurrent decision events to be analyzed in light of similar events underconditions of varying dis-similarity using machine learned criteriaobtained from that previous data; results of this analysis in additionto other factors may be analyzed by an uncertainty estimation module 213to further tune the level of confidence to be included with the finishedanalysis. Confidence level would be a weighted calculation of the randomvariable distribution given to each event analyzed. Prediction of theeffects of at least a portion of the events involved with a businessventure under analysis within a system as complex as anything from themicroenvironment in which the client business operates to more expansivearenas as the regional economy or further, from the perspective ofsuccess of the client business is calculated in dynamic systemsextraction and inference module 214, which use, among other toolsalgorithms based upon Shannon entropy, Hartley entropy and mutualinformation dependence theory.

Of great importance in any business decision or new business venture isthe amount of business value that is being placed at risk by choosingone decision over another. Often this value is monetary but it can alsobe competitive placement, operational efficiency or customerrelationship based, for example: the may be the effects of keeping anolder, possibly somewhat malfunctioning customer relationship managementsystem one more quarter instead of replacing it for $14 million dollarsand a subscription fee. The automated planning service module has theability predict the outcome of such decisions per value that will beplaced at risk using programming based upon the Monte Carlo heuristicmodel 216 which allows a single “state” estimation of value at risk. Itis very difficult to anticipate the amount of computing power that willbe needed to complete one or more of these business decision analyseswhich can vary greatly in individual needs and often are run withseveral alternatives concurrently. The invention is therefore designedto run on expandable clusters 215, in a distributed, modular, andextensible approach, such as, but not exclusively, offerings of Amazon'sAWS. Similarly, these analysis jobs may run for many hours to completionand many clients may be anticipating long waits for simple “what if”options which will not affect their business operations in the near termwhile other clients may have come upon a pressing decision situationwhere they need alternatives as soon as possible. This is accommodatedby the presence of a job queue that allows analysis jobs to beimplemented at one of multiple priority levels from low to urgent. Incase of a change in more hypothetical analysis jobs to more pressing,job priorities can also be changed during run without loss of progressusing the priority based job queue 218.

Structured plan analysis result data may be stored in either a generalpurpose automated planning engine executing Action Notation ModelingLanguage (ANML) scripts for modeling which can be used to prioritizeboth human and machine-oriented tasks to maximize reward functions overfinite time horizons 217 or through the graph-based data store 145,depending on the specifics of the analysis in complexity and time run.

The results of analyses may be sent to one of two client facingpresentation modules, the action outcome simulation module 125 or themore visual simulation capable observation and state estimation module140 depending on the needs and intended usage of the data by the client.

FIG. 3 is a diagram of an exemplary architecture of a system for thecapture and storage of time series data from sensors with heterogeneousreporting profiles according to an embodiment of the invention 300. Inthis embodiment, a plurality of sensor devices 310 a-n stream data to acollection device, in this case a web server acting as a network gateway315. These sensors 310 a-n can be of several forms, some non-exhaustiveexamples being: physical sensors measuring humidity, pressure,temperature, orientation, and presence of a gas; or virtual such asprogramming measuring a level of network traffic, memory usage in acontroller, and number of times the word “refill” is used in a stream ofemail messages on a particular network segment, to name a small few ofthe many diverse forms known to the art. In the embodiment, the sensordata is passed without transformation to the data management engine 320,where it is aggregated and organized for storage in a specific type ofdata store 325 designed to handle the multidimensional time series dataresultant from sensor data. Raw sensor data can exhibit highly differentdelivery characteristics. Some sensor sets may deliver low to moderatevolumes of data continuously. It would be infeasible to attempt to storethe data in this continuous fashion to a data store as attempting toassign identifying keys and then to store real time data from multiplesensors would invariably lead to significant data loss. In thiscircumstance, the data stream management engine 320 would hold incomingdata in memory, keeping only the parameters, or “dimensions” from withinthe larger sensor stream that are pre-decided by the administrator ofthe study as important and instructions to store them transmitted fromthe administration device 312. The data stream management engine 320would then aggregate the data from multiple individual sensors andapportion that data at a predetermined interval, for example, every 10seconds, using the timestamp as the key when storing the data to amultidimensional time series data store over a single swimlane ofsufficient size. This highly ordered delivery of a foreseeable amount ofdata per unit time is particularly amenable to data capture and storagebut patterns where delivery of data from sensors occurs irregularly andthe amount of data is extremely heterogeneous are quite prevalent. Inthese situations, the data stream management engine cannot successfullyuse strictly single time interval over a single swimlane mode of datastorage. In addition to the single time interval method the inventionalso can make use of event based storage triggers where a predeterminednumber of data receipt events, as set at the administration device 312,triggers transfer of a data block consisting of the apportioned numberof events as one dimension and a number of sensor ids as the other. Inthe embodiment, the system time at commitment or a time stamp that ispart of the sensor data received is used as the key for the data blockvalue of the value-key pair. The invention can also accept a raw datastream with commitment occurring when the accumulated stream datareaches a predesigned size set at the administration device 312.

It is also likely that during times of heavy reporting from a moderateto large array of sensors, the instantaneous load of data to becommitted will exceed what can be reliably transferred over a singleswimlane. The embodiment of the invention can, if capture parameterspre-set at the administration device 312, combine the data movementcapacity of two or more swimlanes, the combined bandwidth dubbed ametaswimlane, transparently to the committing process, to accommodatethe influx of data in need of commitment. All sensor data, regardless ofdelivery circumstances are stored in a multidimensional time series datastore 325 which is designed for very low overhead and rapid data storageand minimal maintenance needs to sap resources. The embodiment uses akey-value pair data store examples of which are Riak, Redis and BerkeleyDB for their low overhead and speed, although the invention is notspecifically tied to a single data store type to the exclusion of othersknown in the art should another data store with better response andfeature characteristics emerge. Due to factors easily surmised by thoseknowledgeable in the art, data store commitment reliability is dependenton data store data size under the conditions intrinsic to time seriessensor data analysis. The number of data records must be kept relativelylow for the herein disclosed purpose. As an example one group ofdevelopers restrict the size of their multidimensional time serieskey-value pair data store to approximately 8.64×104 records, equivalentto 24 hours of 1 second interval sensor readings or 60 days of 1-minuteinterval readings. In this development system the oldest data is deletedfrom the data store and lost. This loss of data is acceptable underdevelopment conditions but in a production environment, the loss of theolder data is almost always significant and unacceptable. The inventionaccounts for this need to retain older data by stipulating that ageddata be placed in long term storage. In the embodiment, the archivalstorage is included 330. This archival storage might be locally providedby the user, might be cloud based such as that offered by Amazon WebServices or Google or could be any other available very large capacitystorage method known to those skilled in the art.

Reliably capturing and storing sensor data as well as providing forlonger term, offline, storage of the data, while important, is only anexercise without methods to repetitively retrieve and analyze mostlikely differing but specific sets of data over time. The inventionprovides for this requirement with a robust query language that bothprovides straightforward language to retrieve data sets bounded bymultiple parameters, but to then invoke several transformations on thatdata set prior to output. In the embodiment isolation of desired datasets and transformations applied to that data occurs using pre-definedquery commands issued from the administration device 312 and acted uponwithin the database by the structured query interpreter 335. Below is ahighly simplified example statement to illustrate the method by which avery small number of options that are available using the structuredquery interpreter 335 might be accessed.

SELECT [STREAMING | EVENTS] data_spec FROM [unit] timestamp TO timestamp15 GROUPBY (sensor_id, identifier) FILTER [filter_identifier] FORMAT[sensor [AS identifier] [, sensor [AS identifier]] . . . ] (TEXT | JSON| FUNNEL | KML | GEOJSON | TOPOJSON);

Here “data_spec” might be replaced by a list of individual sensors froma larger array of sensors and each sensor in the list might be given ahuman readable identifier in the format “sensor AS identifier”. “unit”allows the researcher to assign a periodicity for the sensor data suchas second (s), minute (m), hour (h). One or more transformationalfilters, which include but a not limited to: mean, median, variance,standard deviation, standard linear interpolation, or Kalman filteringand smoothing, may be applied and then data formatted in one or moreformats examples of with are text, JSON, KML, GEOJSON and TOPOJSON amongothers known to the art, depending on the intended use of the data.

FIG. 4 is a system diagram illustrating connections between corecomponents of the physical system necessary for agent creation,according to an aspect of the invention. An agent creation engine 420may receive input from either a user 410 or specific algorithms andcomputer input 430, over a network 440 or all operating on the samecomputing device, in order to store desired parameters and settings foragents to be created. Algorithms and computer input 430 which may beused to specify parameters and attributes of agents may be based atleast in part on simulated environments 450, for example a biologicalsimulation may create new entities through reproduction, and agentscreated in the simulation in this manner may inherit some or all oftheir traits and parameters from their “parent” agents. Alternatively,the agent creation engine 420 may have settings in place to give allcreated agents certain default attributes, according to userspecifications. The agent creation engine 420 may create agents inobject text notation such as JSON, or may create them in databaseentries, or other preferred methods of storing data, according to anaspect. An indefinite number of agents 461, 462 may be created in thisway by an agent creation engine 420 and may be stored on a server 460,over a network 440, which may be either the same server as those runningsimulated environments 450 or may be separate from the simulation engineserver 450. In either implementation, an arbitrary number of simulations451, 452, 453 running on a server 450 will communicate either over anetwork 440 with another server 460 to access agent templates andspecifications as needed, or access such information on the samecomputer if such information is held on the same server as the oneexecuting the simulation or simulations.

FIG. 5 is a method diagram illustrating the basic steps in the processof an agent being created. Two primary methods of agent creation exist510, one in which a user or external tool may specify some or alldetails of an agent's characteristics and parameters 511, and one inwhich a simulation may specify the attributes an agent must have 512. Itis possible for a combination of pre-generated or inherited attributes512 to be combined with user-specified attributes 511 for an agent'screation 510 for use in a simulated environment. Such attributes whichmay be specified programmatically or by a user with any variety of userinterfaces may include things such as graphical representation of theagent in a simulation (for example, for use in a video game),reproductive capabilities (for example, in a simulation of a bacteriaand surrounding environment), or other behaviors, attributes, orcharacteristics. These attributes or characteristics must then be storedon the server hosting the agent data 460, 520, using any formatpreferred at the time of execution, including NoSQL databases such asMONGODB™, SQL databases such as MYSQL™, raw or formatted text includingJSON formatted text outside of databases, or other storageimplementations. The agents specified and created in this manner mayinteract with a simulated environment 530, whether such a simulatedenvironment is hosted on the same server 460 or a separate server 450over a network 440. Interactions 530 in the simulated environment aredependent on what specifications the simulated environment and agentsboth possess. For example, it is possible with a hierarchical learningagent model to simulate a basic bacterium or similar single-celledorganism, or a virus, in a petri dish or basic simulated livingorganism, which will be explored further in FIG. 7 .

FIG. 6 is a method diagram illustrating several key steps agents createdmay take in interacting with, and learning from, environments theyoperate in. Agents may take many varied actions dependent on theirspecifications and the specifications of the simulated environment theyare in 610, which may or may not be further dependent or influenced by ahuman user in the environment (for example agents may be left to operatein the environment for research purposes in simulating real-worldenvironments, rather than needing human users to interact with them asin the case of a video game). A key component of agents created by theagent creation engine 420 is that they have some goal, whether definedin the simulated environment or defined in their own specifications andattributes, and work according to achieve this goal in a hierarchicalfashion 620. For example, an agent being used to simulate a male humanlaborer in an economic simulation may have the end-goal of providing aliving for a family represented by other simulated agents, and thesimulated environment may fire the laborer in an effort to explainpossible outcomes of job displacement for certain working groups. Insuch a hypothetical simulation, the hierarchical learning and acting ofthe agent in pursuit of a goal 620 may cause the agent to seek new work,which may require learning new skills, which may require lifestylechanges, which may require time and the spouse or partner of the agentto acquire a different job to pay bills in the meantime, and more. Manydifferent actions and learned behaviors may be exhibited and acquired byindividual agents, differentiating them from each other during asimulation execution, behaving in a non-deterministic environment suchas stochastic game. In this way, agents may learn from each other 630,in an effort to build a knowledge base of how to achieve an outcome in acertain environment, or they may be programmed to not teach each other,and individual agent paths may be examined to examine the efficacy ofeach individual set of actions taken by each agent, according to whatthe purpose of the simulated environment is. Agents in a simulatedenvironment may also learn from a human user in the simulate environment630—for example, adaptive AI techniques in a strategy or tactical videogame may result in agents learning patterns of behavior with the humanuser, and devising new methods of attack or defense in an effort tostump the player. In any such learning instance, regardless of thespecific implementation, the result of the learned behavior is stored ina server 640 for semi-permanency, whether it is stored separately fromthe simulated environments along with the agent data 460 or whether suchdata is stored with the simulate environments on the same server 450,and regardless of which particular data storage implementation ischosen, including raw and formatted text, NoSQL databases, or SQLdatabases.

FIG. 7 is a method diagram illustrating a specific use-case forhierarchical learning agents in varied simulated environments,specifically the case of biological simulations. Agents in this use casemay be created to simulate phages 710, for the goal of learning thebehavior of bacteriophages and virophages (a new term in the medicalfield which may change usage in the future, by which is meant a viruswhich infects other viruses, often giant viruses, as a form of parasiticorganism). Agents used to simulate these phages in behavior may theninfect other cellular agents and viruses, according to how phages behavein the real world 720, with the mechanics of DNA replication andmovement of many bacteria and viruses being fairly well-known andcapable of being simulated. Utilizing hierarchical learning and possiblyeven group-behaviors, agents in this simulation may work to achievebasic goals such as “reproduce maximally” or other basic goals 730 whichmay elucidate the behavior and efficacy of using virophage therapy forpatients, a new idea in medical research which requires research.

FIG. 8 is a method diagram illustrating a specific use-case forhierarchical learning agents in varied simulated environments,specifically the case of economic simulations. Agents in this case maysimulate consumers and/or workers in an economic simulation 810,including purchasing of goods, using those goods up, replacing goods,working to produce goods or receive wages, and spending those wages, inan effort to map an economic environment to track purchases and laborersas the environment shifts, to monitor simulated learning behavior in ashifting economic landscape 830. The rules of the economy and thesimulation itself may be diverse and detailed, or simplistic, dependingon the desired research 820; using a stochastic method for the game, itis possible to run multiple simulations, with many agents, to findmultiple different outcomes for each economy and each individual agent.In such economic simulations, agents and the results in the simulatedenvironment's economy may be monitored for research purposes onanalogues to a real-world economy 840, to isolate trends, find optimalbehaviors in a given economic situation, and more.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (“ASIC”), or on a network interface card.

Software/hardware hybrid implementations of at least some of the aspectsdisclosed herein may be implemented on a programmable network-residentmachine (which should be understood to include intermittently connectednetwork-aware machines) selectively activated or reconfigured by acomputer program stored in memory. Such network devices may havemultiple network interfaces that may be configured or designed toutilize different types of network communication protocols. A generalarchitecture for some of these machines may be described herein in orderto illustrate one or more exemplary means by which a given unit offunctionality may be implemented. According to specific aspects, atleast some of the features or functionalities of the various aspectsdisclosed herein may be implemented on one or more general-purposecomputers associated with one or more networks, such as for example anend-user computer system, a client computer, a network server or otherserver system, a mobile computing device (e.g., tablet computing device,mobile phone, smartphone, laptop, or other appropriate computingdevice), a consumer electronic device, a music player, or any othersuitable electronic device, router, switch, or other suitable device, orany combination thereof. In at least some aspects, at least some of thefeatures or functionalities of the various aspects disclosed herein maybe implemented in one or more virtualized computing environments (e.g.,network computing clouds, virtual machines hosted on one or morephysical computing machines, or other appropriate virtual environments).

Referring now to FIG. 9 , there is shown a block diagram depicting anexemplary computing device 10 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 10 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 10 may be configuredto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 10 includes one or more centralprocessing units (CPU) 12, one or more interfaces 15, and one or morebusses 14 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 12 maybe responsible for implementing specific functions associated with thefunctions of a specifically configured computing device or machine. Forexample, in at least one embodiment, a computing device 10 may beconfigured or designed to function as a server system utilizing CPU 12,local memory 11 and/or remote memory 16, and interface(s) 15. In atleast one embodiment, CPU 12 may be caused to perform one or more of thedifferent types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 12 may include one or more processors 13 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 13 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 10. In a specific embodiment,a local memory 11 (such as non-volatile random access memory (RAM)and/or read-only memory (ROM), including for example one or more levelsof cached memory) may also form part of CPU 12. However, there are manydifferent ways in which memory may be coupled to system 10. Memory 11may be used for a variety of purposes such as, for example, cachingand/or storing data, programming instructions, and the like. It shouldbe further appreciated that CPU 12 may be one of a variety ofsystem-on-a-chip (SOC) type hardware that may include additionalhardware such as memory or graphics processing chips, such as a QUALCOMMSNAPDRAGON™ or SAMSUNG EXYNOS™ CPU as are becoming increasingly commonin the art, such as for use in mobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 15 are provided as network interface cards(NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 15 may forexample support other peripherals used with computing device 10. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radiofrequency (RF), BLUETOOTH™, near-field communications (e.g., usingnear-field magnetics), 802.11 (Wi-Fi), frame relay, TCP/IP, ISDN, fastEthernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) orexternal SATA (ESATA) interfaces, high-definition multimedia interface(HDMI), digital visual interface (DVI), analog or digital audiointerfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 15 may include physical ports appropriate for communicationwith appropriate media. In some cases, they may also include anindependent processor (such as a dedicated audio or video processor, asis common in the art for high-fidelity A/V hardware interfaces) and, insome instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown in FIG. 9 illustrates one specificarchitecture for a computing device 10 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 13 may be used, and such processors 13may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 13 handles communicationsas well as routing computations, while in other embodiments a separatededicated communications processor may be provided. In variousembodiments, different types of features or functionalities may beimplemented in a system according to the invention that includes aclient device (such as a tablet device or smartphone running clientsoftware) and server systems (such as a server system described in moredetail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 16 and local memory 11) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 16 or memories 11,16 may also be configured to store data structures, configuration data,encryption data, historical system operations information, or any otherspecific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a JAVA™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 10 ,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 20 includes processors 21that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 24. Processors 21 may carry out computinginstructions under control of an operating system 22 such as, forexample, a version of MICROSOFT WINDOWS™ operating system, APPLE OSX™ oriOS™ operating systems, some variety of the Linux operating system,ANDROID™ operating system, or the like. In many cases, one or moreshared services 23 may be operable in system 20, and may be useful forproviding common services to client applications 24. Services 23 may forexample be WINDOWS™ services, user-space common services in a Linuxenvironment, or any other type of common service architecture used withoperating system 21. Input devices 28 may be of any type suitable forreceiving user input, including for example a keyboard, touchscreen,microphone (for example, for voice input), mouse, touchpad, trackball,or any combination thereof. Output devices 27 may be of any typesuitable for providing output to one or more users, whether remote orlocal to system 20, and may include for example one or more screens forvisual output, speakers, printers, or any combination thereof. Memory 25may be random-access memory having any structure and architecture knownin the art, for use by processors 21, for example to run software.Storage devices 26 may be any magnetic, optical, mechanical, memristor,or electrical storage device for storage of data in digital form (suchas those described above, referring to FIG. 9 ). Examples of storagedevices 26 include flash memory, magnetic hard drive, CD-ROM, and/or thelike.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 11 , there is shown ablock diagram depicting an exemplary architecture 30 for implementing atleast a portion of a system according to an embodiment of the inventionon a distributed computing network. According to the embodiment, anynumber of clients 33 may be provided. Each client 33 may run softwarefor implementing client-side portions of the present invention; clientsmay comprise a system 20 such as that illustrated in FIG. 10 . Inaddition, any number of servers 32 may be provided for handling requestsreceived from one or more clients 33. Clients 33 and servers 32 maycommunicate with one another via one or more electronic networks 31,which may be in various embodiments any of the Internet, a wide areanetwork, a mobile telephony network (such as CDMA or GSM cellularnetworks), a wireless network (such as Wi-Fi, WiMAX, LTE, and so forth),or a local area network (or indeed any network topology known in theart; the invention does not prefer any one network topology over anyother). Networks 31 may be implemented using any known networkprotocols, including for example wired and/or wireless protocols.

In addition, in some embodiments, servers 32 may call external services37 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 37 may take place, for example, via one or morenetworks 31. In various embodiments, external services 37 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 24 are implemented on a smartphone or other electronicdevice, client applications 24 may obtain information stored in a serversystem 32 in the cloud or on an external service 37 deployed on one ormore of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 33 or servers 32 (or both)may make use of one or more specialized services or appliances that maybe deployed locally or remotely across one or more networks 31. Forexample, one or more databases 34 may be used or referred to by one ormore embodiments of the invention. It should be understood by one havingordinary skill in the art that databases 34 may be arranged in a widevariety of architectures and using a wide variety of data access andmanipulation means. For example, in various embodiments one or moredatabases 34 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, HADOOP CASSANDRA™, GOOGLE BIGTABLE™, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 36 and configuration systems 35. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 36 orconfiguration system 35 or approach is specifically required by thedescription of any specific embodiment.

FIG. 12 shows an exemplary overview of a computer system 40 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 40 withoutdeparting from the broader scope of the system and method disclosedherein. Central processor unit (CPU) 41 is connected to bus 42, to whichbus is also connected memory 43, nonvolatile memory 44, display 47,input/output (I/O) unit 48, and network interface card (NIC) 53. I/Ounit 48 may, typically, be connected to keyboard 49, pointing device 50,hard disk 52, and real-time clock 51. NIC 53 connects to network 54,which may be the Internet or a local network, which local network may ormay not have connections to the Internet. Also shown as part of system40 is power supply unit 45 connected, in this example, to a mainalternating current (AC) supply 46. Not shown are batteries that couldbe present, and many other devices and modifications that are well knownbut are not applicable to the specific novel functions of the currentsystem and method disclosed herein. It should be appreciated that someor all components illustrated may be combined, such as in variousintegrated applications, for example Qualcomm or Samsungsystem-on-a-chip (SOC) devices, or whenever it may be appropriate tocombine multiple capabilities or functions into a single hardware device(for instance, in mobile devices such as smartphones, video gameconsoles, in-vehicle computer systems such as navigation or multimediasystems in automobiles, or other integrated hardware devices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client components.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A system for generating learning agents insimulated environments, comprising: a computing device comprising amemory and a processor; an agent creation engine comprising a firstplurality of programming instructions stored in the memory and operatingon the processor, wherein the first plurality of programminginstructions, when operating on the processor, cause the computingdevice to: receive one or more agent goals; select an agent simulationbased on the one or more agent goals; create a plurality of agents, eachagent being an instance of the agent simulation and having one of theone or more goals; and at least one simulation manager comprising asecond plurality of programming instructions stored in the memory andoperating on the processor, wherein the second plurality of programminginstructions, when operating on the processor, cause the computingdevice to: receive a simulation goal related to the one or more agentgoals; select a dynamic environment simulation based on the simulationgoal; execute the dynamic environment simulation using a plurality ofmeta-models, wherein the plurality of simulation agents used in theexecution is managed according to at least one meta-model, wherein theat least one meta-model comprises a plurality of relationships betweenthe dynamic environment simulation and the agent simulation; andcontinue execution of the dynamic environment simulation that evolveswith agent behavior from the execution of the plurality of agents andthe at least one meta-model until the simulation goal has been reachedor until each agent has achieved at least one agent goal from the one ormore agent goals; wherein each of the plurality of agents takesdifferent actions in a non-deterministic environment based onspecifications of the respective agent and specifications of the dynamicenvironment simulation to achieve the one or more agent goals; whereinthe actions taken by the agents include hierarchical learning accordingto set agent goals, learning patterns of user behavior and learnedbehavior of agents stored from previous simulations; wherein thedifferent actions and learned behaviors acquired by individual agentsfurther differentiating them from each other during the simulationexecution; and examine the outcomes of each individual set of actionstaken by each agent according to the simulation goal.
 2. The system ofclaim 1, wherein the agent creation engine operates on a simulationexecution server.
 3. The system of claim 1, wherein the agent creationengine operates on a separate server from the simulation executionserver.